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Abstract: The value of the duration of participation in a small spacecraft program has 
not previously been sufficiently characterized. This work seeks to determine whether most 
relevant benefits are received by participants quickly (suggesting that participant education 
would be best achieved by shorter duration exposure to multiple domains) or accrues over 
time (suggesting that prolonged work on a single project would be most beneficial). 
The experiences of the student participants in the OpenOrbiter Small Spacecraft 
Development Initiative at the University of North Dakota are analyzed in an attempt to 
answer this question. To this end, correlation between the duration of program participation 
and the level of benefit received (across five categories) is assessed herein. 

Keywords: project-based learning; small spacecraft development; CubeSat; duration of 
participation and learning 


1. Introduction 

With the growing prevalence of small spacecraft development programs as an educational 
experience to teach students about multiple science, technology education and math (STEM) 
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disciplines, it is important to determine what the ideal duration of participation is to maximize 
student benefit. Practically, the answer to the question of whether benefit from participation accrues 
quickly at first and then the rate of additional gains diminishes, whether the rate of gain grows over 
time or whether it remains constant is sought. Alternately, it is possible that benefit is event- or 
experience-driven, meaning that there is no (or limited) correlation between the duration of 
participation and the level of benefit received. Instead benefit is received in chunks via participation in 
key milestones and events. 

This paper begins to answer this question via assessing whether correlation exists between the 
duration of participation and the level of benefit received in each of five metrics (and one combined 
metric) for participants in the University of North Dakota’s OpenOrbiter Small Spacecraft 
Development Initiative (OOSDI). OOSDI is a CubeSat program that aims to demonstrate the efficacy 
of the Open Prototype for Educational NanoSats (OPEN) designs [1], which seek to enable the 
construction of a 1-U CubeSat-class spacecraft for under USD $5,000 [2], The first results from 
an ongoing longitudinal study that will assess the benefits received by participants over time are 
presented herein. 

2. Background 

The work presented herein builds on existing work on project-based learning and small spacecraft 
development. This section presents an overview of relevant prior work on project-based learning, 
small spacecraft development as well as an overview of the OpenOrbiter program and prior work on 
its assessment. 

2.1. Project-based Learning 

Project-based learning (PBL), also known as problem-based learning, is a technique where students 
learn by doing. While the concept is by no means new (as the apprenticeship style of learning has 
been used throughout history [3,4]), it is seen as a departure from the traditional lecture-based style 
of instruction. The benefits are seen by some [5] as so great as to have an effect on national 
competitiveness on an international scale. The development of small spacecraft and CubeSats provides 
students with PBL-style educational benefits [6-8] in their discipline of participation. PBL has been 
shown to be effective in a diverse number of disciplines relevant to small spacecraft development, 
including aerospace [9], computer [10], electrical [11] and mechanical [12] engineering, 
computer science [13], project management [14] and entrepreneurship [15]. It has been shown to be 
effective across a wide range of educational and age levels [8,16]. It has also been shown to 
deliver benefits in addition to driving learning about course topics. These include improved student 
self-image [17], creativity [18], motivation [17], material understanding [19], workforce preparation [19], 
job placement [20] and academic program [21] and knowledge retention [22]. 

In the university context, PBL can occur in several formats. Students may engage in PBL activities 
as part of a regular course, such as a course project [8] or PBL-style course. They may participate as 
part of an independent or directed study [23] or to satisfy a senior design requirement [24]. They may 
also participate for extracurricular educational enrichment [23]. 
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2.2. Small Spacecraft Development 

Small spacecraft come in many varieties. In fact, the exact definition of what is a small spacecraft 
is elusive. Prefixes have been defined [25] to classify types of spacecraft; however, there is no line 
defining where small ends and larger sizes begin. Swartwout [26,27] proffers that size isn’t the 
defining attribute. Instead, he suggests that so-called “university class” spacecraft should be defined by 
their educational missions, risk tolerance and the ability to serve as a testing bed for out-of-the-box 
concepts. The CubeSat is one form factor that is commonly used for university-class spacecraft. 
Developed initially by Bob Twiggs and Jordi Puig-Suari as a tool to facilitate aerospace engineering 
education [28], CubeSats are now widely used by education [29,30] as well as being developed 
for science [31-34], government [35], military [36,37] and commercial [38,39] purposes. 
Their development is being aided [40] by the availability of free-to-developer launch services from the 
U.S. Air Force [41], NASA [42] and the ESA [43]. Lower-cost commercial launches are also on the 
horizon [44,45]. Low-cost development approaches, such as OPEN [2], are also enabling adoption 
via reducing the cost of spacecraft development. In 2013, 30 academic and 50 non-academic 
CubeSats have been manifested and over 100 institutions have participated in the development of 
a CubeSat-class spacecraft [30]. 

2.3. Related Remote Sensing Activities 

Spacecraft are one mechanism for conducting remote sensing. Several other remote sensing 
educational activities also bear mention. Rundquist and Vandeberg [46] showed how in-the-field 
exercises on the ground could be utilized to drive student understanding, learning and excitement, 
without the need for a launch. Saad [47] and Jackson, et al. [48], conversely, showed how a low-cost 
(compared to a satellite launch) high altitude balloon launch could generate excitement for K-12 
students and college undergraduates. Nordlie, et al. [49] demonstrated how an even lower-cost solar 
balloon (which can only reach 60,000 feet, as compared to the approximate 100,000 foot ceiling of 
helium high altitude balloons) can be used to generate similar excitement for under ten-percent of 
the cost. Mountrakis, et al. [50] quantified the impact of a project-based remote sensing activity on 
student performance on post-project exercises, showing a possible impact on the performance of 
lower-achieving students. They also noted qualitative benefits. 

2.4. The OpenOrbiter Program 

The OpenOrbiter Small Spacecraft Development Initiative commenced in 2012, following a 
thematically-related program. Its goals, name and logo are all student-developed. It has involved both 
STEM and non-STEM students from across the campus of the University of North Dakota [8]. 
OpenOrbiter seeks to demonstrate the efficacy of the designs [1] of the Open Prototype for Educational 
NanoSats via development, launch and on-orbit operations of an OPEN-based spacecraft. 
OPEN facilitates the low-cost development of CubeSats via making complete designs, software, 
fabrication and testing instructions and other materials publically available. With OPEN, a CubeSat 
can be developed for a parts cost (excluding payload-specific components) of approximately $5,000 [2]. 
This is significantly less than the $40,000 or more that might be spent buying a one-time-use kit-based 
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spacecraft or the $250,000 cost of developing the designs from scratch [40]. These lower cost levels will 
allow greater penetration of spacecraft development or spacecraft-based experiments into the educational 
systems of more affluent countries and enable spacecraft development in less-affluent ones [51]. 

2.5. Assessment of the Educational Vcdue of the OpenOrbiter Program 

Prior work on the assessment of the educational value of the OpenOrbiter program and the benefits 
it provides has been conducted [23]. The level of increase in student self-identified status in five metric 
areas was reported. These increases were also determined to be attributed to program participation by 
student participants. It was shown that comparative levels of overall benefit were enjoyed by both 
graduate and undergraduate students and this improvement was attributed to program participation. 
Team leads were shown to receive more (approximately double the level of) benefit from participation 
and those participating more were also shown to receive greater benefit. Some correlation between 
time participating (based on surveys taken at one time which recorded data about individuals that had 
participated for different lengths of time) and benefit level was also demonstrated. The percentage of 
participants in each category showing improvement was shown to have minimal correlation with the 
amount of time that individuals had been in the program. Excluding some outlying data, the level of 
attribution of benefit to the program was consistent across the participation durations. Only very 
limited correlation between grade-level, GPA and benefits was identified. Other work [52] identified 
the benefits that students sought from their participation in the program and quantified the level of 
interest among participants in receiving these benefits. This article expands this analysis to focus on 
the impact that the duration of participation has on the level of benefit received. 

3. Experimental Goals and Design 

Five key areas of focus were initially identified when attempting to ascertain the educational 
effectiveness of the OOSDI. These were technical skills in the area of focus, spacecraft design skills, 
space excitement, and skills and comfort of giving presentations. While additional areas of focus have 
subsequently been identified, only one survey has been conducted characterizing participant attainment 
of benefits in these areas, so they are not considered at this time. 

Initially, a 26-question survey [23] was conducted. The first 12 questions collected demographic 
data about the participant and their involvement in OOSDI. This included information about their 
academic status (undergraduate vs. graduate, class, time in program, GPA). They were also asked 
about their participation in OOSDI (number of years participating, hours participating per week and 
whether they participated for academic credit or not). They were also asked whether they had previous 
involvement in spacecraft design and, if so, the type of involvement. 

The second version of this survey asked 42 questions, including the 26 from the initial survey. 
The duration of participation was changed to semesters from years to avoid causing students confusion 
in answering this question. The data from the earlier survey was multiplied by two (as UND has 
two normal-year semesters and the program has had extremely minimal participation by students over 
the summer months) to allow the year and semester numbers to be utilized together. This treatment is 
consistent with the clarification provided regarding how to answer the previous year’s question. 
An additional option was added to the question regarding academic participation and the questions 
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regarding whether the participant had received credit from participating and the type of for-credit 
participation were combined into a single question (including answer in the format “yes—type” and the 
answer “no”). 

The majority of the survey is comprised of questions presented on a 9-point scale. In all cases 9 is 
the superior answer (indicating greater experience/knowledge/etc.) 5 is neutral and 1 indicates the 
inferior answer. Students were asked, on both surveys, to characterize their pre-participation status and 
current status with regards to each of the five metrics. 

The data from these two survey forms has been utilized to assess the correlation between the 
duration of participation (in semesters) and the level of benefit attained. This level of benefit has been 
calculated in all cases by subtracting the pre-participation status level from the post-participation status 
level. In a very limited number of cases, this resulted in inexplicable negative values. These have been 
investigated (as discussed in [23]) and replaced with zeroes, as they appear (based on responses to the 
attribution questions) to represent clerical errors by participants; they are, otherwise, uninterpretable. 

4. Results and Discussion 

This section presents the process and results of the analysis of the data that was collected as 
described in the foregoing section. In addition to the five key metrics for which data was collected, 
an additional aggregate metric was created by adding the benefit attained from each of the other areas 
together. As a participant would not necessarily receive benefit in all areas, this combined metric may 
serve as more holistic view of the value to the participant. This analysis process commenced with the 
correlation between all respondents’ performance in each of the five key metrics and the combined 
aggregate metric. This data, which is based on 31 respondent surveys including nine masters’ students, 
twenty-one undergraduate students and one individual who did not respond as to his or her students 
status, is presented in Table 1. This data included responses from seventeen non-lead participants, 
thirteen lead-level participants and one individual who did not respond with regards to this question. 
Note that all analysis was performed in Microsoft Excel. 


Table 1. Correlation between number of semesters involved in the program and the 
five assessed metrics. 


Technical 

Spacecraft 

Level of 

Presentation 

Presentation 

Aggregate 

Skill 

Design 

Excitement 

Skills 

Comfort 

Improvement 

0.26 

0.17 

- 0.13 

0.24 

0.21 

0.22 


From the data presented, it is clear that there is limited correlation between the metrics and the 
amount of time spent participating, when neglecting all other confounding factors. For four of the 
metrics, correlation levels of between 0.17 and 0.26 are reported; one (spacecraft design) reports a 
negative correlation level. The aggregate improvement correlation level is also below 0.25. 
As correlation levels are on a -1 to 1 scale (with -1 indicating perfect inverse correlation and 
1 indicating perfect correlation), these values do not provide much support to the thesis that prolonged 
participation provides greater levels of benefit. 
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Because of the nature of student participation, however, some students will receive benefits at 
different rates. Possible confounding variables are now considered. In Table 2, the correlation process 
is separated into graduate or undergraduate status. 


Table 2. Comparison of the correlation between number of semesters involved in the 
program and the five assessed metrics, for masters and undergraduate students. 



Technical 

Spacecraft 

Level of 

Presentation 

Presentation 

Aggregate 


Skill 

Design 

Excitement 

Skills 

Comfort 

Improvement 

Bachelors Students 

-0.12 

-0.21 

-0.24 

0.10 

0.04 

-0.17 

Masters Students 

0.85 

0.84 

0.35 

0.63 

0.37 

0.89 


For the masters’ students, this data shows strong correlation in three areas (technical skills, 
spacecraft design and aggregate improvement), moderate correlation is also shown in presentation 
skills. The level of excitement and presentation comfort metrics, show greater correlation levels 
than with the non-separated data. However, for the bachelors’ students, limited positive and negative 
correlation data is still presented. For the purposes of assessing statistical significance, an improvement 
as a function of duration of participation value was created by dividing each value by the duration 
(in semesters) of participation. From this, the difference in terms of space excitement (0.04) was 
significant at p = 0.05 and technical skills (0.08) and aggregate improvement (0.07) were significant at 
p = 0.10. Spacecraft design (0.18), presentation skills (0.12) and presentation confidence (0.48) were 
not shown to be significantly different at either p = 0.05 or p = 0.10. 

Next, the correlation assessment is performed separating participants and team leads. This data is 
presented in Table 3. 


Table 3. Comparison of the correlation between the number of semesters involved in the 
program and the five assessed metrics between those serving in a participant and 
team lead role. 



Technical 

Spacecraft 

Level of 

Presentation 

Presentation 

Aggregate 


Skill 

Design 

Excitement 

Skills 

Comfort 

Improvement 

Participant 
—Bachelors 

-0.12 

-0.24 

-0.26 

0.27 

0.19 

-0.17 

Participant 

—Masters 

1 

1 

N/A 

N/A 

1 

1 

Participant 
—Combined 

-0.09 

-0.19 

-0.23 

0.28 

0.21 

-0.11 

Team Lead 

—Bachelors 

-0.11 

0.00 

-0.49 

-0.66 

-0.32 

-0.34 

Team Lead 

—Masters 

0.83 

0.86 

0.27 

0.57 

0.32 

0.88 

Team Lead 

—Combined 

0.72 

0.63 

-0.03 

0.13 

0.21 

0.52 


Team leads, overall, show moderate correlation in technical skills, spacecraft design and aggregate 
improvement and limited correlation in presentation skills and comfort. More pronounced results are 
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demonstrated by the masters-level students. Very limited negative correlation is shown for the level of 
excitement. Undergraduate participants, however, show limited positive or negative correlation for all 
metrics, while the very limited data set for masters-level participants shows perfect correlation. 

As the amount of weekly involvement in the project may affect the level of benefits obtained, 
the five metrics are now correlated separated by the number of hours worked each week (one 8+ response 
has been removed as it represented insufficient data for analysis). This data is presented in Table 4. 


Table 4. Comparison of the correlation between the number of semesters involved in the 
program and the five assessed metrics between those devoting between 1-3.99 h per week 
and those devoting 4-7.99 h per week. 



Technical 

Skill 

Spacecraft 

Design 

Level of 

Excitement 

Presentation 

Skills 

Presentation 

Comfort 

Aggregate 

Improvement 

1-3.99—Bachelors 

0.02 

-0.19 

-0.14 

0.31 

0.19 

-0.04 

1-3.99—Masters 

0.87 

0.82 

0.26 

0.50 

0.86 

0.87 

1-3.99—Combined 

0.30 

0.08 

-0.27 

0.36 

0.39 

0.27 

4-7.99—Bachelors 

-0.53 

-0.33 

-0.54 

-0.56 

-0.25 

-0.60 

4—7.99—Masters 

1.00 

1.00 

1.00 

1.00 

N/A 

1.00 

4-7.99—Combined 

0.33 

0.32 

-0.05 

0.04 

-0.24 

0.18 


While this segmentation produces 5 combined correlation values in the 0.3 to 0.4 range and 3 in the 
+0.2-0.3 range, no clear trend emerges for the combined data. The masters’ students, however, 
show strong correlation in the majority of the metrics. The limited set of undergraduate 
high-commitment data appears to show a pronounced negative correlation; however, this is likely 
attributable to students entering and leaving this group between survey administrations (instead of the 
unrealistic conclusion that students were unlearning skills). It also may be indicative of students 
re-evaluating their own skill levels in light of a better understanding of the subject material. 
Next, the data was correlated segmented by whether the participants received academic credit for their 
participation or not. This data is presented in Table 5. 


Table 5. Comparison of the correlation between the number of semesters involved in the 
program and the five assessed metrics between those participating for course credit and 
those not participating for course credit. 



Technical 

Skill 

Spacecraft 

Design 

Level of 

Excitement 

Presentation 

Skills 

Presentation 

Comfort 

Aggregate 

Improvement 

Credit—Bachelors 

-0.23 

-0.32 

-0.08 

0.08 

0.25 

-0.24 

Credit—Combined 

0.38 

0.30 

-0.04 

0.19 

-0.25 

0.23 

No Credit—Bachelors 

-0.25 

-0.62 

-0.40 

-0.13 

-0.32 

-0.48 

No Credit—Masters 

0.79 

0.82 

0.63 

0.53 

-0.35 

0.82 

No Credit—Combined 

0.03 

-0.14 

-0.12 

0.12 

0.43 

0.10 


Again, several combined correlation values in the 0.3 to 0.4 range are produced and one value in the 
0.4—0.5 range is generated. Two values in the +0.2 to 0.3 range are also indicated. However, no moderate 
or strong correlation values are indicated. Insufficient masters-level for-credit participants were present 
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to allow reporting; however, the impact of this limited set on the combined metric is pronounced. 
There is also a clear difference between the masters and bachelors-level students in the no-credit category. 

The majors of the participants may also have some impact on the correlation between the duration 
of participation and the level of benefit received. As the project is run out of the UND Computer 
Science Department, there has been consistent Computer Science student involvement throughout the 
project. Thus, correlation is now performed for only Computer Science students. This data is presented 
in Table 6. 


Table 6. Correlation between number of semesters involved in the program and the 
five assessed metrics, for computer science students. 


Technical 

Spacecraft 

Level of 

Presentation 

Presentation 

Aggregate 

Skill 

Design 

Excitement 

Skills 

Comfort 

Improvement 

Computer Science 0.31 

0.14 

-0.21 

0.18 

0.17 

0.20 


Again, no strong trends are present. One correlation value in the 0.3 to 0.4 range and two in the 
±0.2 to 0.3 range are produced. Finally, the correlation segmented by both the field of major 
(divided into computer science and others) and academic level (graduate or undergraduate) is 
calculated. This data is presented in Table 7. 


Table 7. Correlation between the number of semesters participating and increase in the 
five assessed metrics, for computer science and non-computer science students at both the 
bachelors’ and masters’ levels. 



Technical 

Skill 

Spacecraft 

Design 

Level of 

Excitement 

Presentation 

Skills 

Presentation 

Comfort 

Aggregate 

Improvement 

Bachelors 







—Computer Science 

-0.07 

-0.22 

-0.27 

0.08 

0.01 

-0.17 

Masters 







—Computer Science 

0.95 

0.88 

0.07 

0.42 

0.34 

0.95 

Bachelors—Other 

-0.95 

-0.90 

N/A 

N/A 

N/A 

-1.00 

Masters—Other 

0.90 

0.89 

0.58 

N/A 

0.58 

0.86 


This data indicates very strong correlation for both computer science and non-computer science 
masters-level students in the technical skill, spacecraft design and aggregate improvement categories. 
Moderate correlation is shown for non-computer science students in level of excitement and 
presentation comfort. The masters-level computer science students also produce one correlation value 
in the 0.3 to 0.4 range and one in the 0.4 to 0.5 range. Note that in several cases, data characteristics 
generated a divide-by-zero issue for the excel correlation function. These are indicated with a “N/A” 
in Table 7. 

What is problematic in this data is the very strong negative correlation values reported for 
non-computer science undergraduates (-0.95, -0.9 and -1). As it seemed unlikely that this could be 
attributable to the nature of the program (e.g., students are gaining less value the longer they spend), 
this was investigated. A limited number of respondents and some respondents that achieved significant 
gain in a single semester as compared to other (distinctly different) respondents that achieved moderate 





Ecluc. Sci. 2014 , 4 


149 


levels of improvement over a longer period of time is attributable for this. This result is, thus, 
a quirk of the data and limited number of respondents in this category. 

5. Conclusions and Future Work 

This paper has considered the correlation between the duration of participation in the program and 
the level of value attained. For several groups of participants, this correlation was shown to exist. 
However, it was not shown to exist in the general case (attributable, as demonstrated, to the presence 
of confounding variables). There were also some groups where no correlation between benefit level 
and participation duration could be demonstrated. For graduate students, a very strong correlation level 
was shown to exist in the technical and spacecraft design skills categories. Strong correlation was 
shown when considering the master’s students as a group and this was even stronger when they were 
divided into computer science and non-computer science students. Presentation skills was shown to 
have a moderate level of correlation as a combined (all master’s students) group. This correlation 
couldn’t be assessed separately for the non-computer science master’s students and was not as 
pronounced when only the computer science master’s students were considered. Level of excitement 
and presentation comfort showed moderate correlation for the non-computer science masters students, 
and a lower level of correlation for the computer science master’s students. Both the computer science 
and non-computer science master’s students showed a strong (0.95 in the case of computer science 
master’s students) aggregate improvement correlation; this was also evidenced in the combined 
(all master’s students) data. 

Team leads (which included both graduate and undergraduates) showed moderate correlation 
between the duration of participation and level of value attained in three areas (technical and spacecraft 
design skills) and aggregate improvement. However, beyond the team leads, no grouping could be 
found from the data elements collected that removed a confounding variable to demonstrate strong 
positive correlation between the duration of participation and the level of benefit attained. This could 
indicate that undergraduates do not gain significantly in additional benefit beyond some level of 
participation. It could also be a function of limited sample sizes. Perhaps there is some other 
confounding variable that was not identified or surveyed that, if the data was segmented by it, 
would allow this correlation to be demonstrated. Continued analysis of this topic will be a subject for 
future work. 

The presence or absence of correlation is important for several reasons. First, the presence of 
correlation suggests the value of longer duration participation. This correlation was an expected 
outcome. Second, when data shows a significant difference when segmented over a variable, 
this identifies this is a factor in the level of benefit that students attain. Some of these (such as team 
lead status, for-credit participation status or the number of hours worked a week) can be controlled 
if greater levels of benefit are seen for one status than others. Others (such as the major or graduate 
versus undergraduate status) demonstrate what groups should be targeted for participation, as students 
in these groups receive particular benefit. The identification of groups (of all types) that do not perform 
as well as others also serves to focus attention on improving (or identifying non-modifiable limiting 
factors) the outcomes of the program for these groups. Continued analysis of what levels of benefit are 
generated and for what groups of students will serve as a subject for ongoing work. When this study 
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was started, individual identifying information was not collected on the surveys, limiting their utility 
for individual-granularity-level long-term tracking. This, combined with the high level of churn 
(particularly with undergraduate participants, both overall and entering and exiting particular 
classifications), has resulted in data that seems to show a negative correlation with skill learning 
(which makes little sense). This same trend is not present in the masters-level students where 
participation has been more consistent (though attrition has been significant, movement in and out of 
classifications has not). Future work will therefore include the collection of identifying information to 
facilitate individual tracking. 
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